Using Reinforcement Learning to Spider the Web Eeciently

Consider the task of exploring the Web in order to nd pages of a particular kind or on a particular topic. This task arises in the construction of search engines and Web knowledge bases. This paper argues that the creation of eecient web spiders is best framed and solved by reinforcement learning, a branch of machine learning that concerns itself with optimal sequential decision making. One strength of reinforcement learning is that it provides a formalism for measuring the utility of actions that give beneet only in the future. We present an algorithm for learning a value function that maps hyperlinks to future discounted reward by using naive Bayes text classiiers. Experiments on two real-world spidering tasks show a threefold improvement in spider-ing eeciency over traditional breadth-rst search, and up to a twofold improvement over reinforcement learning with immediate reward only.